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" (57) Abstract: A system, (i.e., a method, an apparatus, and computer-executable process steps), providing sharpness enhancement 
for a coded video, in which a usefulness metric, calculates how much a pixel can be enhanced without increasing coding artifacts. 
The usefulness metric is separate from the enhancement algorithm such that a variety of different enhancement algorithms can be 
)^ used in conjunction with the metric. 
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Method and system for sharpness enhancement for coded video 



This invention uses the UME of co-pending application, Apparatus and 
Method for Providing a Usefulness Metric based on Coding Information for Video 
Enhancement, inventors Lilla Boroczky and Johan Janssen, filed concurrently herewith. The 
present invention is entitled to the benefit of Provisional Patent Application Serial Number 
5 60/260,845 filed January 10, 2001 . 

The present invention is directed to a system and method for enhancing the 
sharpness of encoded/transcoded digital video, without enhancing encoding artifacts, which 
has particular utility in connection with spatial domain sharpness enhancement algorithms 
used in multimedia devices. 

10 The development of high-quality multi-media devices, such as set-top boxes, 

high-end TV's, Digital TV's, Personal TV's, storage products, PDA's, wireless internet 
devices, etc., is leading to a variety of architectures and to more openness towards new 
features for these devices. Moreover, the development of these new products and their ability 
to display video data in any format, has resulted in new requirements and opportunities with 

15 respect to video processing and video enhancement algorithms. Most of these devices receive 
and/or store video in the MPEG-2 format and in the future they may receive/store in the 
MPEG-4 format. The picture quality of these MPEG sources can vary between very good and 
extremely bad. 

Next generation storage devices, such as the blue-laser-based Digital Video 
20 Recorder (DVR) will have to some extent HD (ATSC) capability and are an example of the 
type of device for which a new method of picture enhancement would be advantageous. An 
HD program is typically broadcast at 20 Mb/s and encoded according to the MPEG-2 video 
standard. Taking into account the approximately 25 GB storage capacity of the DVR, this 
represents about a two-hour recording time of HD video per disc. To increase the record 
25 time, several long-play modes can be defined, such as Long-Play (LP) and Extended-Long- 
Play (ELP) modes. 

For LP-mode the average storage bitrate is assumed to be approximately 10 
Mb/s, which allows double record time for HD. As a consequence, transcoding is an integral 
part of the video processing chain, which reduces the broadcast bitrate of 20 Mb/s to the 
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storage bitrate of 10 Mb/s. During the MPEG-2 transcoding, the picture quality (e.g., 
sharpness) of the video, is most likely reduced. However, especially for the LP mode, the 
picture quality should not be compromised too much. Therefore, for the LP mode, post- 
processing plays an important role in improving the perceived picture quality. 
5 To date, most of the state-of-the-art sharpness enhancement algorithms were 

developed and optimized for analog video transmission standards like NTSC, PAL and 
SECAM. Traditionally, image enhancement algorithms either reduce certain unwanted 
aspects in a picture (e.g., noise reduction) or improve certain desired characteristics of an 
image (e.g., sharpness enhancement). For these emerging storage devices, the traditional 

10 sharpness enhancement algorithms may perform sub-optimally on MPEG encoded or 
transcoded video due to the different characteristics of these sources. In the closed video 
processing chain of the storage system, information which allows for determining the quality 
of the encoded source can be derived from the MPEG stream. This information can 
potentially be used to increase the performance of image enhancement algorithms. 

1 5 Because image quality will remain a distinguishing factor for high-end video 

products, new approaches for performing image enhancement, specifically adapted for use 
with these sources, will be beneficial. In C-J Tsai, P. Karunaratne, N. P. Galatsanos and A. K. 
Katsaggelos, "A Compressed Video Enhancement Algorithm", Proc. of IEEE, ICIP'99, 
Kobe, Japan, Oct. 25-28, 1999, the authors propose an iterative algorithm for enhancing 

20 video sequences that are encoded at low bit rates. For MPEG sources, the degradation of the 
picture quality originates mostly from the quantization function. Thus, the iterative gradient- 
projection algorithm employed by the authors uses coding information such as quantization 
step size, macroblock types and forward motion vectors in its cost function. The algorithm 
shows promising results for low bit rate video, however its main disadvantage is its high 

25 computational complexity. 

In B. Martins and S. Forchammer, "Improved Decoding of MPEG-2 Coded 
Video", Proc. ofIBC'2000, Amsterdam, The Netherlands, Sept. 7-12, 2000, pp. 109-115, the 
authors describe a new concept for improving the decoding of MPEG-2 coded video. 
Specifically, a unified approach for deinterlacing and format conversion, integrated in the 

30 decoding process, is proposed. The technique results in considerably higher picture quality 
than that obtained by ordinary decoding. However, to date, its computational complexity 
prevents its implementation in consumer applications. 

Both papers describe video enhancement algorithms using MPEG coding 
information and a cost function. However, both of these scenarios, in addition to being 
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impractical, combine the enhancement and the cost function. A cost function determines how 
much, and at which locations in a picture, enhancement can be applied. The problem which 
results from this combination of cost and enhancement functions is that only one algorithm 
can be used with the cost function. 
5 The present invention addresses the foregoing needs by providing a system, 

(i.e., a method, an apparatus, and computer-executable process steps), in which a usefulness 
metric, calculates how much a pixel can be enhanced without increasing coding artifacts. 

It is an object of this invention to provide a system in which the usefulness 
metric is separate from the enhancement algorithm such that a variety of different 
10 enhancement algorithms can be used in conjunction with the metric. 

It is a further object of the invention to provide a usefulness metric which can 
be tuned towards the constraints of the system such that an optimal trade-off between 
performance and complexity is assured. 

It is a further object of the invention to provide a system of image 
1 5 enhancement which will perform optimally with encoded and transcoded video sources. 

This brief summary has been provided so that the nature of the invention may 
be understood quickly. A more complete understanding of the invention can be obtained by 
reference to the following detailed description of the preferred embodiments thereof in 
connection with the attached drawings. 

20 

For a better understanding of the invention, reference is made to the following 

drawings: 

Fig. 1 is a block diagram of the invention 
25 Fig. 2 is a flowchart of the invention using only the coding gain 



Fig. 1 shows a system in which the present invention can be implemented, for 
example, in a video receiver 56. Figure 1 illustrates how a usefulness metric (UME) can be 
30 applied to, a sharpness enhancement algorithm, adaptive peaking, for example. (Other 

sharpness enhancement algorithms, besides adaptive peaking, can also be used.) The adaptive 
peaking algorithm, directed at increasing the amplitude to the transient of a luminance signal 
2, does not always provide optimal video quality for an a priori encoded/transcoded video 
source. This is mainly a result of the fact that the characteristics of the MPEG source are not 
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taken into account. In the present invention, a UME is generated, which does take into 
account the characteristics of the MPEG source. The example algorithm, adaptive peaking, is 
extended to use this UME, thereby increasing the performance of the algorithm significantly. 

The adaptive peaking algorithm and the principle of adaptive peaking, are well 
5 known in the prior art. An example is shown in Fig. 1 . The algorithm includes four control 
blocks, 6 8 10 12. These pixel-based control blocks 681012 operate in parallel and each 
calculate a maximum allowable gain factor gl g2 g3 g4, respectively, to achieve a target 
image quality. These control blocks 6 8 10 12 take into account particular local 
characteristics of the video signal such as contrast, dynamic range, and noise level, but not 

10 coding properties. The coding gain block 14 uses the usefulness metric (UME) 1 8 to 

determine the allowable amount of peaking g CO din g 36. A dynamic gain control 16 selects the 
minimum of the gains gl 28, g2 30, g3 32, g4 34, which is added to the gcoding generating a 
final gain g 38. The multiplier 22, multiplies the final gain 38 by the high-pass signal 20, 
which has been filtered by the 2D peaking filter 4. The adder 24 adds this product to the 

15 original luminance value of a pixel 2. In this maimer, the enhanced luminance signal 26 is 
generated. 

The UME 18 calculates on a pixel by pixel basis, how much a pixel or region 
can be enhanced without increasing coding artifacts. The UME 18 is derived from the MPEG 
coding information present in the bitstream. 
20 Choosing the MPEG information to be used with the UME 1 8 is far from 

trivial. The information must provide an indication of the spatio-temporal characteristics or 
picture quality of the video. 

The finest granularity of MPEG information, which can be directly obtained 
during decoding is either block-based or macroblock-based. However for spatial (pixel) 
25 domain video enhancement, the UME 1 8 must be calculated for each pixel of a picture in 
order to ensure the highest picture quality. 

One parameter easily extracted from MPEG information is the quantization 
parameter, as it is present for every coded macroblock (MB). The higher the quantization 
parameter, the coarser the quantization, and therefore, the higher the quantization error. A 
30 high quantization error results in coding artifacts and consequently, enhancement of pixels in 
a MB with a high quantization parameter must be suppressed more. 

Another parameter that can easily be extracted from the MPEG stream is the 
number of bits spent in coding a MB or block. The value of the aforementioned coding 
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information is dependent upon other factors including: scene content, bitrate, picture type, 
and motion estimation/compensation. 

Both the quantization parameter and the number of bits spent are widely used 
in rate control calculations of MPEG encoding and are commonly used to calculate the 
5 coding complexity. Coding complexity is defined as the product of the quantization 
parameter and the number of bits spent to encode a MB or block. Coding complexity is 
therefore described by the following equation: 

complMB/biock(k ? l)=mquant(k ? l) * bitSMBftio C k(k,l) 
where mquant is the quantization parameter and bitSMB/biock is the number of bits of DCT 
10 coefficients used to encode the MB or block(k,l). The underlying assumption is that the 

higher the complexity of a MB or block with respect to the average complexity of a frame, 
the higher the probability of having coding artifacts in that MB or block. Thus, enhancement 
should be suppressed for the pixels of the blocks with relatively high coding complexity. 

Accordingly, the UME 1 8 of pixel(i j) can be defined by the following 

15 equation: 

UME(ij) = 1 - complpi X ei(i j)/2 * compl 
where compl P i xe i(i j) is the coding complexity of pixel (i,j) and compl is the average coding 
complexity of a picture. In the present invention, compl P i xe i(i j) is estimated from the MB or 
block complexity map Figure 2 48 by means of bilinear interpolation Figure 2 58. 

20 In one aspect of the invention, UME(i j) can range from 0 to 1 . In this aspect, 

zero means that no sharpness enhancement is allowed for a particular pixel, while 1 means 
that the pixel can be freely enhanced without the risk of enhancing any coding artifacts. 

The UME equation can be extended, by the addition of a term directly related 
to the quantization parameter, to incorporate a stronger bitrate dependency. This can be 

25 especially advantageous for video that has been encoded at a low bitrate. 

For skipped or uncoded MBs/blocks, the UME is estimated Figure 2 50 from surrounding 
values. 

Because the UME 18 is calculated to account for coding characteristics, it only 
prevents the enhancement of coding artifacts such as blocking and ringing. Thus, the 
30 prevention or reduction of artifacts of non-coding origin, which might result from applying 
too much enhancement, is addressed by other parts of the sharpness enhancement algorithm. 

The aforementioned UME 18 can be combined with any peaking algorithm, or 
it can be adapted to any spatial domain sharpness enhancement algorithm. It is also possible 
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to utilize coding information Figure 2 46 and incorporate scene content related information 
Figure 2 44, in combination with an adaptive peaking algorithm. 

In this embodiment, shown in Figure 2, the four control blocks 6 8 10 12 
shown in Figure 1 are eliminated. Scene content information, such as edge information 44, is 
5 incorporated into the coding gain calculation via the edge detection 42. The scene-content 
related information 44 compensates for the uncertainty of the UME calculation Fig. 118, the 
uncertainty resulting from assumptions made and interpolations applied in its calculation, 
Fig. 2 58 36. 

In this embodiment, the coding gain of a pixel (ij) 36 is determined by 
10 summing the UME which is embedded in the coding gain calculation 36 with an Edge Map 
44 related term according to the equation below: 

gcoding(i ,j)=UME(ij) + g ed ge(i j) 

UME is defined above and g e d ge is based on edge-related pixel information. 

It should be noted that the complexity map 56 of the MB/block has an 
1 5 inherited block structure. To decrease this non-desirable characteristic of the complexity map 
56, a spatial low-pass filtering 52 is applied by a filter. An example filter kernel, which can 
be used for low-pass filtering is: 

"111" 



LP 



compI_map = 



111 
111 



Another problem is that abrupt frame to frame changes in the coding gain for 
20 any given pixel can result in temporally inconsistent sharpness enhancement, which is 

undesirable. Such changes can also intensify temporally visible and annoying artifacts such 
as mosquito noise. 

To remedy this effect, temporal filtering 54 is applied to the coding gain using 
the gain of the previous frame. To reduce the high computational complexity and memory 
25 requirement, instead of filtering the gain-map, the MB or block-based complexity map 48 is 
filtered temporally using an IIR filter 54. The following equation represents this processing: 
complMB/biock(r,s,t) = k * complMB/biock(r,s,t) + 

seal * (1-k) * complMB/biock(r,s,t-l) 
where r,s is the spatial coordinate of a MB or block, t represents the current picture, k is the 
30 IIR filter coefficient and seal is a scaling term taking into account the complexity differences 
among different picture types. The coding gain 36 is then applied to the adaptive peaking 
algorithm using the frame 1 60 to produce an enhanced frame 1 60. 
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The invention can also be applied to HD and SD sequences such as would be 
present in a video storage application having HD capabilities and allowing long-play mode. 
The majority of such video sequences are transcoded to a lower storage bitrate from 
broadcast MPEG-2 bitstreams. For the long play mode of this application, format change can 
5 also take place during transcoding. Well-known SD video sequences encoded, decoded, and 
then processed with the sharpness enhancement algorithm, according to the present 
invention, provide superior video quality for a priori encoded or transcoded video sequences 
as compared to algorithms that do not use coding information. 

The present invention has been described with respect to particular illustrative 
10 embodiments. It is to be understood that the invention is not limited to the above-described 
embodiments and modifications thereto, and that various changes and modifications may be 
made by those of ordinary skill in the art without departing from the spirit and scope of the 
appended claims. 
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1 . A method for enhancing image quality comprising: 

- developing a usefulness metric 18 which identifies a limit to sharpness 
enhancement that can be applied to decoded video without enhancing coding artifacts; and 

" applying the usefulness metric 1 8 to at least one sharpness enhancement 
5 algorithm 4, the usefulness metric 18 and the sharpness enhancement algorithm 4 being 
separate such that the usefulness metric 18 can be used with a variety of algorithms. 

2. A method for enhancing the sharpness of a coded digital video, comprising the 
steps of: 

10 - selecting and extracting statistical information 46 from a coded video bit 

stream in order to identify the video's coding complexity 48; 

- based upon the coding complexity 48, developing a usefulness metric 18 for 
the coded video, which identifies a limit to sharpness enhancement that can be applied to the 
coded video after it is decoded, without enhancing coding artifacts; and 

15 - applying a sharpness enhancement algorithm 4 to the decoded video to 

increase sharpness within the limit prescribed by the usefulness metric 18. 

3. The method as claimed in claim 2 wherein the sharpness enhancement 
algorithm is a peaking algorithm 4. 

20 

4. The method as claimed in claim 2 wherein the sharpness enhancement 
algorithm is a spatial-domain algorithm 4. 

5. The method as claimed in claim 2 wherein the usefulness metric 18 is 
25 calculated on a pixel-by-pixel basis. 

6. The method as claimed in claim 2 wherein the coding complexity 48 is defined 
as the product of a quantization parameter and a number of bits used to code a macro block. 
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7. The method as claimed in claim 2 wherein the coding complexity 48 is defined 
as the product of a quantization parameter and a number of bits used to code a block. 

8. The method as claimed in claim 2, wherein the usefulness metric 1 8 occupies 
5 a range, a first terminus of the range meaning no sharpness enhancement is allowed for a 

particular pixel and second terminus of the range meaning that the pixel can be freely 
enhanced. 



9. The method as claimed in claim 2, wherein the method is also applied to 
1 0 skipped macroblocks, the usefulness metric 1 8 being estimated based upon the coding 

complexity 48 of surrounding macro blocks or the coding complexity 48 of a previous frame. 

10. The method as claimed in claim 2, wherein the method is also applied to 
uncoded blocks, the usefulness metric 18 being estimated based upon the coding complexity 

15 48 of surrounding blocks or the coding complexity 48 of a previous frame. 

1 1 . The method as claimed in claim 2, wherein in addition to the usefulness metric 
18, scene-content related information 42 is incorporated into a coding gain calculation. 

20 12. The method as claimed in claim 2, wherein the scene-content related 

information 44 is derived from edge 42 information. 

1 3 . The method as claimed in claim 5, wherein coding gain 1 4 of a pixel is 

determined by the equation: 
25 gcoding(i j) = UME(ij) + gedge(ij) 

and wherein i and j are pixel coordinates, g CO din g 36 is the pixel coding gain, UME 18 is the 
usefulness metric and g e d ge is based upon edge-related information 44 derived from the 
image. 

30 14. The method as claimed in claim 13, wherein spatial low-pass filtering 52 is 

applied to a complexity map 48 calculated from the coded digital video. 



1 5 . The method as claimed in claim 13, wherein temporal filtering 54 is applied to 

the coding gain 36 using the coding 36 gain of a previous frame. 
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16. The method as claimed in claim 13, wherein the equation 14 can be extended 

to include an additional term directly related to the quantization parameter. 

5 17. The method as claimed in claim 6, wherein a block-based complexity map 48 

is filtered temporally using an IIR filter 54. 

18. The method as claimed in claim 6 ? wherein a macro block-based complexity 
map 48 is filtered temporally using an IIR filter 54. 

10 

19. The method as claimed in claim 17 or 18, wherein the temporal filtering 54 is in 
accordance with the following equation: 

compl MB/biock (r ? s,t) = k * compl MB/biock (r,s,t) + seal * (1-k) * compl MB/biock (r,s,t-l) 
and wherein r,s is the spatial coordinate of a macro block or block, 1 60 represents the current 
1 5 picture, k is the IIR filter coefficient and seal is a scaling term taking into account picture 
complexity 48 determined by the image's picture type. 

20. A device for image quality enhancement comprising: 

- a peaking filter 4 which filters a decoded luminance 2 signal, generating a 
20 high pass signal 20; 

- a plurality of pixel based control blocks 68, 10, 12, 14, operating in parallel 
on the decoded luminance signal 2, each calculating a maximum allowable gain factor, based 
upon a characteristic of the luminance signal, wherein at least one control block is a coding 
gain block 14 which implements a usefulness metric 18 which determines the allowable 

25 amount of peaking; 

- a dynamic gain control 32 for selecting a minimum gain based upon the 
calculated maximum gain factors; 

- a multiplier 22 for multiplying the high pass signal 20 by the minimum gain 
38 generating a multiplied signal; and 

30 - an adder 24 for combining the decoded luminance signal 2 with the 

multiplied signal, generating an enhanced signal 26. 



21. 



A device as claimed in claim 20, wherein the control blocks comprise: 
- a contrast control block 6; 
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- a dynamic range control block 8; 

- a clipping prevention control block 10; 

- an adaptive coring control block 12; and 

- a coding gain block 14, all of the blocks being connected in parallel. 

5 

22. A device for enhancing the image quality of a digital video comprising: 

- a usefulness metric 1 8 generator which identifies a limit to sharpness 
enhancement that can be applied, without enhancing coding artifacts, to decoded digital 
video; 

10 - a controller which applies the usefulness metric to at least one sharpness 

enhancement algorithm 4, the usefulness metric 18 and the sharpness enhancement algorithm 
4 being separate such that the usefulness metric can be used with a variety of algorithms. 

23. A system which enhances sharpness of a coded digital video, comprising: 
15 - a selector which selects and extracts statistical information from a coded 

video bit stream in order to identify the video's coding complexity 48; 

- a usefulness metric generator that, based upon the coding complexity 50, 
develops a usefulness metric 1 8 for the coded digital video after decoding, which identifies a 
limit to sharpness enhancement that can be applied to a decoded video without enhancing 

20 coding artifacts; and 

- a sharpness enhancer 4 which applies a sharpness enhancement algorithm to 
the decoded video to increase sharpness within the limit prescribed by the usefulness metric. 

24. Computer-executable process steps to enhance image quality, the computer- 
25 executable process steps being stored on a computer-readable medium and comprising: 

- an extracting step to extract statistical information 46 from a coded video bit 
stream in order to identify a video's coding complexity; 

- a generating step to generate a usefulness metric 1 8 for a coded video based 
upon the coding complexity 48, which identifies a limit to sharpness enhancement that can be 

30 applied to the coded video after decoding without enhancing coding artifacts; and 

- an enhancement step to enhance the sharpness of the image by applying a 
sharpness enhancement algorithm 4 to a decoded video to increase sharpness within the limit 
prescribed by the usefulness metric 18. 
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25. Means for enhancing the sharpness of a coded digital video, comprising: 

- extracting means for extracting statistical information 46 from a coded video 
bit stream in order to identify the coded digital video's coding complexity 48; 

- generating means for developing a usefulness metric 18 for the coded digital 
5 video, based upon the coding complexity 48, which identifies a limit to sharpness 

enhancement that can be applied to the coded digital video after decoding without enhancing 
coding artifacts; and 

- enhancement means for applying a sharpness enhancement algorithm 4 to a 
decoded video to increase sharpness within the limit prescribed by the usefulness metric. 

10 

26. A signal, embodied in a carrier wave, representing data for enhancing 
sharpness of a decoded digital video, comprising: 

- statistical information 46 selected from a coded video bit stream to be used in 
identifying the complexity 48 of a video; 

15 - a usefulness metric 1 8, based upon the complexity 48 of the video, which 

identifies a limit to sharpness enhancement which can be applied to the decoded video 
without enhancing coding artifacts; and 

- a sharpness enhancement algorithm 4 to be used for increasing the sharpness 
of the decoded video within the limit prescribed by the usefulness metric. 

20 

27. A method for enhancing image quality comprising the steps of: 

- peaking filtering 4 a coded luminance signal 2, increasing the amplitude of 
the luminance signal 2 and generating a high pass signal 20; 

- calculating at least one maximum gain factor 928, 930, 932, 934, 936 for the 
25 luminance signal, based on a characteristic of the luminance signal, wherein at least one gain 

factor calculation 14 implements a usefulness metric 18 which determines an allowable 
amount of peaking which will not intensify coding artifacts; 

- selecting a minimum gain 938 from the maximum gain factors 
928,930,932,934,936; 

30 - multiplying the high pass signal 20 by the minimum gain 938 generating a 

multiplied signal 22; and 

- adding 24 a decoded luminance signal with the multiplied signal, generating 
an enhanced signal 26. 
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28. A video receiving device 56 comprising: 

- a peaking filter 4 which filters a decoded luminance signal, generating a high 

pass signal 20; 

- a plurality of pixel based control blocks 6,8,10,12,14, operating in parallel on 
5 the decoded luminance signal 2, each calculating a maximum allowable gain factor 

928,930,932,934,936, based upon a characteristic of the luminance signal 2, wherein at least 
one control block is a coding gain block 14 which implements a usefulness metric 18 which 
determines the allowable amount of peaking; 

- a dynamic gain control 16 for selecting a minimum gain 938 based upon the 
1 0 calculated maximum gain factors; 

- a multiplier 22 for multiplying the high pass signal 20 by the minimum gain 
938 generating a multiplied signal 22; and 

- an adder 24 for combining the decoded luminance signal 2 with the 
multiplied signal, generating an enhanced signal 26. 
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